{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Intro to Federated Learning\n",
    "\n",
    "In the last section, we learned about PointerTensors, wey create the underlying infrastructure wey we need for privacy preserving Deep Learning. In this section we go see how we fit use these basic tools to implement our first privacy preserving deep learning algorithm, Federated Learning.\n",
    "\n",
    "Person wey write am:\n",
    "- Andrew Trask - Twitter: [@iamtrask](https://twitter.com/iamtrask)\n",
    "\n",
    "Person wey translate am:\n",
    "- Temitọpẹ Ọladokun - Twitter: [@techie991](https://twitter.com/techie991)\n",
    "\n",
    "\n",
    "### Wetin be Federated Learning?\n",
    "\n",
    "Na the simple, powerful way wey we fit train Deep Learning models. If you dey think about training data, you go get the result from collection process.  People (via devices) dey generate data by recording events wey dey our world. Normally, we go join the data togeda make dey turn one tin, central location so that he go fit train machine learning model. Federated Learning go turn this for him head!\n",
    "\n",
    "Instead make we bring training data to the model (a central server), you go bring the model to the training data (wherever he dey) \n",
    "\n",
    "The idea go allow any person wey dey create the data to get the permanent copy, and you got fit get control over person wey get access to am. He make sense, eh?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 2.1 - A Toy Federated Learning Example\n",
    "\n",
    "Make we start to train a toy model the centralized way. This is about a simple as the models get. We go need: \n",
    "\n",
    "- a toy dataset\n",
    "- a model\n",
    "- some basic training logic for training a model to fit the data.\n",
    "\n",
    "Not: If you no sabi this API - go check [fast.ai](http://fast.ai) and make you take the course before you go continue this tutoria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# A Toy Dataset\n",
    "data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]], requires_grad=True)\n",
    "target = torch.tensor([[0],[0],[1],[1.]], requires_grad=True)\n",
    "\n",
    "# A Toy Model\n",
    "model = nn.Linear(2,1)\n",
    "\n",
    "def train():\n",
    "    # Training Logic\n",
    "    opt = optim.SGD(params=model.parameters(),lr=0.1)\n",
    "    for iter in range(20):\n",
    "\n",
    "        # 1) erase previous gradients (if they exist)\n",
    "        opt.zero_grad()\n",
    "\n",
    "        # 2) make a prediction\n",
    "        pred = model(data)\n",
    "\n",
    "        # 3) calculate how much we missed\n",
    "        loss = ((pred - target)**2).sum()\n",
    "\n",
    "        # 4) figure out which weights caused us to miss\n",
    "        loss.backward()\n",
    "\n",
    "        # 5) change those weights\n",
    "        opt.step()\n",
    "\n",
    "        # 6) print our progress\n",
    "        print(loss.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Na so we get am! We go train basic model for conventional manner. All our data go dey inside our local machine and we go use am do updates to our model.  Federated Learning, no dey work like that. So, make we add one or two tins on the example to do Federated Learning!\n",
    "\n",
    "So, wetin we go need:\n",
    "\n",
    "- create a couple workers\n",
    "- get pointers to training data on each worker\n",
    "- updated training logic to do federated learning\n",
    "\n",
    "    New Training Steps:\n",
    "    - send model to correct worker\n",
    "    - train on the data located there\n",
    "    - get the model back and repeat with next worker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import syft as sy\n",
    "hook = sy.TorchHook(torch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a couple workers\n",
    "\n",
    "bob = sy.VirtualWorker(hook, id=\"bob\")\n",
    "alice = sy.VirtualWorker(hook, id=\"alice\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A Toy Dataset\n",
    "data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]], requires_grad=True)\n",
    "target = torch.tensor([[0],[0],[1],[1.]], requires_grad=True)\n",
    "\n",
    "# get pointers to training data on each worker by\n",
    "# sending some training data to bob and alice\n",
    "data_bob = data[0:2]\n",
    "target_bob = target[0:2]\n",
    "\n",
    "data_alice = data[2:]\n",
    "target_alice = target[2:]\n",
    "\n",
    "# Iniitalize A Toy Model\n",
    "model = nn.Linear(2,1)\n",
    "\n",
    "data_bob = data_bob.send(bob)\n",
    "data_alice = data_alice.send(alice)\n",
    "target_bob = target_bob.send(bob)\n",
    "target_alice = target_alice.send(alice)\n",
    "\n",
    "# organize pointers into a list\n",
    "datasets = [(data_bob,target_bob),(data_alice,target_alice)]\n",
    "\n",
    "opt = optim.SGD(params=model.parameters(),lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train():\n",
    "    # Training Logic\n",
    "    opt = optim.SGD(params=model.parameters(),lr=0.1)\n",
    "    for iter in range(10):\n",
    "        \n",
    "        # NEW) iterate through each worker's dataset\n",
    "        for data,target in datasets:\n",
    "            \n",
    "            # NEW) send model to correct worker\n",
    "            model.send(data.location)\n",
    "\n",
    "            # 1) erase previous gradients (if they exist)\n",
    "            opt.zero_grad()\n",
    "\n",
    "            # 2) make a prediction\n",
    "            pred = model(data)\n",
    "\n",
    "            # 3) calculate how much we missed\n",
    "            loss = ((pred - target)**2).sum()\n",
    "\n",
    "            # 4) figure out which weights caused us to miss\n",
    "            loss.backward()\n",
    "\n",
    "            # 5) change those weights\n",
    "            opt.step()\n",
    "            \n",
    "            # NEW) get model (with gradients)\n",
    "            model.get()\n",
    "\n",
    "            # 6) print our progress\n",
    "            print(loss.get()) # NEW) slight edit... need to call .get() on loss\\\n",
    "    \n",
    "# federated averaging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Well Done!\n",
    "\n",
    "And voilà! We don dey train simple Deep Learning model using Federated Learning!  We go send the model to each worker, generate a new gradient, and then we go bring the gradient back to our local server where we go update our global model. For this process, we no go eva see or request access to the training data wey we dey use! We go preserve Bob and Alice privacy!!!\n",
    "\n",
    "## Shortcomings of this Example\n",
    "\n",
    "As we see sey this example na nice introduction to Federated Learning, it still get him own wahala. The one wey we sabi na, when we call `model.get()`and receive the updated model from Bob or Alice, we go learn plenti tins about Bob and Alice's training data if we look their gradients. We go fit restore their training data in some cases!\n",
    "\n",
    "Wetin we fit do? First startegy people sabi na to **average the gradient across multiple individuals before uploading it to the central server**. This startegy, go require make we use sophisticated PointerTensor objects. So, in the next section, we go learn about more advanced pointer functionality and then we go fit upgrade this Federated Learning example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations!!! - Oya Join the Community!\n",
    "\n",
    "Clap for una sef as you don finish this notebook tutorial! If you enjoy am and you wan join the movement towards privacy preserving, decentralized ownership of AI and the AI supply chain (data), follow the steps wey dey below.\n",
    "\n",
    "### Star PySyft on GitHub\n",
    "\n",
    "The easiset way to helep our community na to star the GitHub repos! This go helep raise awareness of the tools we dey build.\n",
    "\n",
    "- [Star PySyft](https://github.com/OpenMined/PySyft)\n",
    "\n",
    "### Join our Slack!\n",
    "\n",
    "To follow up bumper to bumper on how latest advancements, join our community! You can do so by filling out the form at [http://slack.openmined.org](http://slack.openmined.org)\n",
    "\n",
    "### Join a Code Project!\n",
    "\n",
    "The best way to contribute to our community na to become code contributor! You fit go to PySyft GitHub Issues page and filter for \"Projects\". E go show you all the top level Tickets giving an overview of what projects you fit join! If you no wan join any project, but you wan code small, you fit look for more \"one off\" mini-projects by searching for GitHub issues marked \"good first issue\"\n",
    "\n",
    "- [PySyft Projects](https://github.com/OpenMined/PySyft/issues?q=is%3Aopen+is%3Aissue+label%3AProject)\n",
    "- [Good First Issue Tickets](https://github.com/OpenMined/PySyft/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n",
    "\n",
    "### Donate\n",
    "\n",
    "If you no get time to contribute to our codebase, but still like to lend support, you fit be a Backer on our Open Collective. All donations wey we get na for our web hosting and other community expenses such as hackathons and meetups! meetups!\n",
    "\n",
    "[OpenMined's Open Collective Page](https://opencollective.com/openmined)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
